{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prompt Optimization Techniques\n",
    "\n",
    "## Overview\n",
    "\n",
    "This tutorial explores advanced techniques for optimizing prompts when working with large language models. We focus on two key strategies: A/B testing prompts and iterative refinement. These methods are crucial for improving the effectiveness and efficiency of AI-driven applications.\n",
    "\n",
    "## Motivation\n",
    "\n",
    "As AI language models become more sophisticated, the quality of prompts used to interact with them becomes increasingly important. Optimized prompts can lead to more accurate, relevant, and useful responses, enhancing the overall performance of AI applications. This tutorial aims to equip learners with practical techniques to systematically improve their prompts.\n",
    "\n",
    "## Key Components\n",
    "\n",
    "1. **A/B Testing Prompts**: A method to compare the effectiveness of different prompt variations.\n",
    "2. **Iterative Refinement**: A strategy for gradually improving prompts based on feedback and results.\n",
    "3. **Performance Metrics**: Ways to measure and compare the quality of responses from different prompts.\n",
    "4. **Practical Implementation**: Hands-on examples using OpenAI's GPT model and LangChain.\n",
    "\n",
    "## Method Details\n",
    "\n",
    "1. **Setup**: We'll start by setting up our environment with the necessary libraries and API keys.\n",
    "\n",
    "2. **A/B Testing**: \n",
    "   - Define multiple versions of a prompt\n",
    "   - Generate responses for each version\n",
    "   - Compare results using predefined metrics\n",
    "\n",
    "3. **Iterative Refinement**:\n",
    "   - Start with an initial prompt\n",
    "   - Generate responses and evaluate\n",
    "   - Identify areas for improvement\n",
    "   - Refine the prompt based on insights\n",
    "   - Repeat the process to continuously enhance the prompt\n",
    "\n",
    "4. **Performance Evaluation**:\n",
    "   - Define relevant metrics (e.g., relevance, specificity, coherence)\n",
    "   - Implement scoring functions\n",
    "   - Compare scores across different prompt versions\n",
    "\n",
    "Throughout the tutorial, we'll use practical examples to demonstrate these techniques, providing learners with hands-on experience in prompt optimization.\n",
    "\n",
    "## Conclusion\n",
    "\n",
    "By the end of this tutorial, learners will have gained:\n",
    "1. Practical skills in conducting A/B tests for prompt optimization\n",
    "2. Understanding of iterative refinement processes for prompts\n",
    "3. Ability to define and use metrics for evaluating prompt effectiveness\n",
    "4. Hands-on experience with OpenAI and LangChain libraries for prompt optimization\n",
    "\n",
    "These skills will enable learners to create more effective AI applications by systematically improving their interaction with language models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "First, let's import the necessary libraries and set up our environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "\n",
    "from langchain_openai import ChatOpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "import numpy as np\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "load_dotenv()\n",
    "\n",
    "# Set up OpenAI API key\n",
    "os.environ[\"OPENAI_API_KEY\"] = os.getenv('OPENAI_API_KEY')\n",
    "\n",
    "# Initialize the language model\n",
    "llm = ChatOpenAI(model=\"gpt-4o\")\n",
    "\n",
    "# Define a helper function to generate responses\n",
    "def generate_response(prompt):\n",
    "    \"\"\"Generate a response using the language model.\n",
    "\n",
    "    Args:\n",
    "        prompt (str): The input prompt.\n",
    "\n",
    "    Returns:\n",
    "        str: The generated response.\n",
    "    \"\"\"\n",
    "    return llm.invoke(prompt).content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A/B Testing Prompts\n",
    "\n",
    "Let's start with A/B testing by comparing different prompt variations for a specific task."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Evaluating response based on engagement...\n",
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Evaluating response based on engagement...\n",
      "Prompt A score: 8.33\n",
      "Prompt B score: 9.00\n",
      "Winning prompt: B\n"
     ]
    }
   ],
   "source": [
    "# Define prompt variations\n",
    "prompt_a = PromptTemplate(\n",
    "    input_variables=[\"topic\"],\n",
    "    template=\"Explain {topic} in simple terms.\"\n",
    ")\n",
    "\n",
    "prompt_b = PromptTemplate(\n",
    "    input_variables=[\"topic\"],\n",
    "    template=\"Provide a beginner-friendly explanation of {topic}, including key concepts and an example.\"\n",
    ")\n",
    "\n",
    "# Updated function to evaluate response quality\n",
    "def evaluate_response(response, criteria):\n",
    "    \"\"\"Evaluate the quality of a response based on given criteria.\n",
    "\n",
    "    Args:\n",
    "        response (str): The generated response.\n",
    "        criteria (list): List of criteria to evaluate.\n",
    "\n",
    "    Returns:\n",
    "        float: The average score across all criteria.\n",
    "    \"\"\"\n",
    "    scores = []\n",
    "    for criterion in criteria:\n",
    "        print(f\"Evaluating response based on {criterion}...\")\n",
    "        prompt = f\"On a scale of 1-10, rate the following response on {criterion}. Start your response with the numeric score:\\n\\n{response}\"\n",
    "        response = generate_response(prompt)\n",
    "        # show 50 characters of the response\n",
    "        # Use regex to find the first number in the response\n",
    "        score_match = re.search(r'\\d+', response)\n",
    "        if score_match:\n",
    "            score = int(score_match.group())\n",
    "            scores.append(min(score, 10))  # Ensure score is not greater than 10\n",
    "        else:\n",
    "            print(f\"Warning: Could not extract numeric score for {criterion}. Using default score of 5.\")\n",
    "            scores.append(5)  # Default score if no number is found\n",
    "    return np.mean(scores)\n",
    "\n",
    "# Perform A/B test\n",
    "topic = \"machine learning\"\n",
    "response_a = generate_response(prompt_a.format(topic=topic))\n",
    "response_b = generate_response(prompt_b.format(topic=topic))\n",
    "\n",
    "criteria = [\"clarity\", \"informativeness\", \"engagement\"]\n",
    "score_a = evaluate_response(response_a, criteria)\n",
    "score_b = evaluate_response(response_b, criteria)\n",
    "\n",
    "print(f\"Prompt A score: {score_a:.2f}\")\n",
    "print(f\"Prompt B score: {score_b:.2f}\")\n",
    "print(f\"Winning prompt: {'A' if score_a > score_b else 'B'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iterative Refinement\n",
    "\n",
    "Now, let's demonstrate the iterative refinement process for improving a prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Evaluating response based on engagement...\n",
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Warning: Could not extract numeric score for informativeness. Using default score of 5.\n",
      "Evaluating response based on engagement...\n",
      "Prompt A score: 8.67\n",
      "Prompt B score: 6.67\n",
      "Winning prompt: A\n",
      "Iteration 1 prompt: Explain {topic} in simple terms, covering the different types of approaches such as supervised, unsupervised, and reinforcement learning. Include real-world applications to illustrate its impact, and describe the learning process, including data training and model evaluation. Discuss its benefits, limitations, and challenges, and provide technical insights into algorithms and data preprocessing techniques for a well-rounded understanding.\n",
      "Iteration 2 prompt: Create a comprehensive explanation of {topic} tailored for a specific audience level (beginner, intermediate, or advanced). Clearly define the audience in your response. Discuss the different approaches, such as supervised, unsupervised, and reinforcement learning, and illustrate real-world applications across various industries to demonstrate its impact. Describe the learning process, including data training and model evaluation, and highlight recent advancements or trends in the field. Address the benefits, limitations, and challenges, including ethical considerations and environmental impacts. Provide technical insights into algorithms and data preprocessing techniques, and incorporate visual aids or diagrams to clarify complex concepts. Include interactive elements or exercises, such as a simple coding task, to engage learners. Offer a glossary of key terms and suggest additional resources, like books or online courses, for further exploration of the topic.\n",
      "Iteration 3 prompt: Create an engaging and educational explanation of {topic} specifically designed for beginners. Clearly define the learning objectives at the outset, such as explaining basic concepts, identifying types, and understanding simple algorithms within {topic}. Use simple language and relatable analogies to ensure accessibility. Integrate visual aids like diagrams or flowcharts to depict key ideas, such as different learning approaches or data processing steps, catering to visual learners. Highlight real-world examples to illustrate the practical impact of {topic}, such as applications in technology or daily life scenarios. Incorporate interactive elements that do not require extensive programming knowledge, like using online tools or exploring datasets, to help learners experiment with the concepts. Expand the glossary with easy-to-understand definitions and include links to further explanations or videos. Recommend supplementary materials, such as videos, articles, and podcasts, to suit diverse learning styles. Address common misconceptions about {topic} and include a section on ethical considerations, providing concrete examples and mitigation strategies. Include a feedback mechanism to gather input from readers for continuous improvement of the guide.\n",
      "\n",
      "Final refined prompt:\n",
      "Create an engaging and educational explanation of {topic} specifically designed for beginners. Clearly define the learning objectives at the outset, such as explaining basic concepts, identifying types, and understanding simple algorithms within {topic}. Use simple language and relatable analogies to ensure accessibility. Integrate visual aids like diagrams or flowcharts to depict key ideas, such as different learning approaches or data processing steps, catering to visual learners. Highlight real-world examples to illustrate the practical impact of {topic}, such as applications in technology or daily life scenarios. Incorporate interactive elements that do not require extensive programming knowledge, like using online tools or exploring datasets, to help learners experiment with the concepts. Expand the glossary with easy-to-understand definitions and include links to further explanations or videos. Recommend supplementary materials, such as videos, articles, and podcasts, to suit diverse learning styles. Address common misconceptions about {topic} and include a section on ethical considerations, providing concrete examples and mitigation strategies. Include a feedback mechanism to gather input from readers for continuous improvement of the guide.\n"
     ]
    }
   ],
   "source": [
    "def refine_prompt(initial_prompt, topic, iterations=3):\n",
    "    \"\"\"Refine a prompt through multiple iterations.\n",
    "\n",
    "    Args:\n",
    "        initial_prompt (PromptTemplate): The starting prompt template.\n",
    "        topic (str): The topic to explain.\n",
    "        iterations (int): Number of refinement iterations.\n",
    "\n",
    "    Returns:\n",
    "        PromptTemplate: The final refined prompt template.\n",
    "    \"\"\"\n",
    "    current_prompt = initial_prompt\n",
    "    for i in range(iterations):\n",
    "        try:\n",
    "            response = generate_response(current_prompt.format(topic=topic))\n",
    "        except KeyError as e:\n",
    "            print(f\"Error in iteration {i+1}: Missing key {e}. Adjusting prompt...\")\n",
    "            # Remove the problematic placeholder\n",
    "            current_prompt.template = current_prompt.template.replace(f\"{{{e.args[0]}}}\", \"relevant example\")\n",
    "            response = generate_response(current_prompt.format(topic=topic))\n",
    "        \n",
    "        # Generate feedback and suggestions for improvement\n",
    "        feedback_prompt = f\"Analyze the following explanation of {topic} and suggest improvements to the prompt that generated it:\\n\\n{response}\"\n",
    "        feedback = generate_response(feedback_prompt)\n",
    "        \n",
    "        # Use the feedback to refine the prompt\n",
    "        refine_prompt = f\"Based on this feedback: '{feedback}', improve the following prompt template. Ensure to only use the variable {{topic}} in your template:\\n\\n{current_prompt.template}\"\n",
    "        refined_template = generate_response(refine_prompt)\n",
    "        \n",
    "        current_prompt = PromptTemplate(\n",
    "            input_variables=[\"topic\"],\n",
    "            template=refined_template\n",
    "        )\n",
    "        \n",
    "        print(f\"Iteration {i+1} prompt: {current_prompt.template}\")\n",
    "    \n",
    "    return current_prompt\n",
    "\n",
    "# Perform A/B test\n",
    "topic = \"machine learning\"\n",
    "response_a = generate_response(prompt_a.format(topic=topic))\n",
    "response_b = generate_response(prompt_b.format(topic=topic))\n",
    "\n",
    "criteria = [\"clarity\", \"informativeness\", \"engagement\"]\n",
    "score_a = evaluate_response(response_a, criteria)\n",
    "score_b = evaluate_response(response_b, criteria)\n",
    "\n",
    "print(f\"Prompt A score: {score_a:.2f}\")\n",
    "print(f\"Prompt B score: {score_b:.2f}\")\n",
    "print(f\"Winning prompt: {'A' if score_a > score_b else 'B'}\")\n",
    "\n",
    "# Start with the winning prompt from A/B testing\n",
    "initial_prompt = prompt_b if score_b > score_a else prompt_a\n",
    "refined_prompt = refine_prompt(initial_prompt, \"machine learning\")\n",
    "\n",
    "print(\"\\nFinal refined prompt:\")\n",
    "print(refined_prompt.template)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing Original and Refined Prompts\n",
    "\n",
    "Let's compare the performance of the original and refined prompts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Evaluating response based on engagement...\n",
      "Evaluating response based on clarity...\n",
      "Evaluating response based on informativeness...\n",
      "Evaluating response based on engagement...\n",
      "Original prompt score: 8.67\n",
      "Refined prompt score: 9.00\n",
      "Improvement: 0.33 points\n"
     ]
    }
   ],
   "source": [
    "original_response = generate_response(initial_prompt.format(topic=\"machine learning\"))\n",
    "refined_response = generate_response(refined_prompt.format(topic=\"machine learning\"))\n",
    "\n",
    "original_score = evaluate_response(original_response, criteria)\n",
    "refined_score = evaluate_response(refined_response, criteria)\n",
    "\n",
    "print(f\"Original prompt score: {original_score:.2f}\")\n",
    "print(f\"Refined prompt score: {refined_score:.2f}\")\n",
    "print(f\"Improvement: {(refined_score - original_score):.2f} points\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
